Crawling the Hidden Web Resources: A Review

An ever-increasing amount of information on the Web today is available only through search interfaces. The users have to type in a set of keywords in a search form in order to access the pages from certain Web sites. These pages are often referred to as the Hidden Web or the Deep Web. Since there are no static links to the Hidden Web pages, search engines cannot discover and index such pages and thus do not return them in the results. However, according to recent studies, the content provided by many Hidden Web sites is often of very high quality and can be extremely valuable to many users. In this paper, the study is made how an effective Hidden Web crawler can be built that can autonomously discover and download pages from the Hidden Web. Since the only “entry point” to a Hidden Web site is a query interface, the main challenge that a Hidden Web crawler has to face is how to automatically generate meaningful queries to issue to the site. Here a theoretical framework to investigate the query generation problem is provided for the Hidden Web and effective policies for generating queries automatically have been studied. The policies proceed iteratively, issuing a different query in every iteration. The effectiveness of these policies is experimentally evaluated on 4 real Hidden Web sites and the results are very promising. For instance, in one experiment, one of the policies downloaded more than 90% of a Hidden Web site (that contains 14 million documents) after issuing fewer than 100 queries.